Skip to content

fix(FIX-2166): a held notice reaches its filer soon after a crash ends the filing request - #3834

Merged
jhoffner merged 6 commits into
mainfrom
fix/FIX-2166
Oct 24, 2026
Merged

jhoffner merged 6 commits into
mainfrom
fix/FIX-2166

Conversation

@jhoffner

@jhoffner jhoffner commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

If the server dies while a worker's filing request is still running, a task ending that request filed does not reach the worker for a long time. After the restart the request still reads in_progress. Each delivery of the notice is turned away for it ({ owed: false }, noticeDeferred … whileRunning). The stale-request sweep marks it interrupted at least 60 s later. Then the notice waits again for its next resend, up to 5 minutes later. The FIX-2087 closure saw this in leg b in 1 of 6 runs.

The delay was bounded, not a loss. With the defaults, it was about 90 s plus one resend backoff.

What this does

  • A dead request reads as stopped. waits.running answers false for an in_progress request whose registry heartbeat has been silent past the host's stale-request threshold. This is the same threshold the sweep uses, so a request on a live peer (a BullMQ worker) still reads as running. Some flows can't tell from the heartbeat: heartbeats off (heartbeatIntervalMs: 0), or beating slower than half the threshold. Those keep today's answer. An entry with no heartbeat recorded also keeps today's answer.
  • The pass comes forward when the sweep marks the request. When the stale-request sweep or a startup sweep marks a request interrupted, it moves the wakes kept in that request's session forward to now. The filer's pass then sends the held notice at once, not at its backoff.

With the defaults, a held notice is heard 60 to 90 s after the request's last heartbeat, plus about 2 s. Because a request beats every 10 s, that is about 50 to 92 s after the kill.

What's asked of you

Nothing here is yours to decide. The owner accepted a bound of about a minute for now. The owner-liveness lease that would make this a few seconds is filed as a separate follow-up.

Parts worth reviewing closely

  • packages/engine/src/execution/request-recovery.ts + durability/request-wakes.ts bringSessionWakesForward: the engine doesn't know which internal entry is the filer's pass, so it reads one wake for each internal entry the session's flow declares. For each wake kept there that is due later, it writes one owe, built from that wake. The owe keeps the earlier due time and bumps the version, so a pass in flight settles stale rather than undoing the bring-forward. A wake whose entry is not the pass fires early. Each entry pays only what is due, then moves its wake on, so firing early costs one extra pass and never acts early.
  • packages/engine/src/durability/wait-chain.ts running: a request with no registry entry still answers from its record. That is no proof its process is gone.
  • The threshold is per process, as the sweep's is. A process configured tighter than the one running the request, or a deploy that slows the heartbeat, can read a live request as stopped. Its notice then runs beside that live filer, ahead of the filer's own wait. That is the ordering FIX-2123's hold exists to prevent. The notice is still heard once: takeNotice and waits.ended gate the take. The sweep would reap that request under the same misreading.
  • packages/engine/src/context/liveness-gate.ts heartbeatStaleThreshold: the same rule as the gate's second arm. It reads a registry that isn't shared across processes too. It deliberately ignores the third arm: with sweeping off, this read is the only thing that frees the notice.

Links

Linear FIX-2166. Closure: #3692 (FIX-2087, epic FIX-2056). Merges to main once reviewed.

How to review this

How to review this. This is a bug fix on the direct route: there is no spec and no spec PR, because the hard part was the diagnosis and it happened in code. This PR is the only gate, so unlike a spec-backed change, the approach is in scope: nothing here was signed off upstream.

Most valuable here: the diagnosis chain. Does the repro actually reproduce the reported failure? Is the named cause the real one? Does the fix address that cause rather than the symptom? Does the regression test fail without the fix? Then the second path: the legacy shape, the null boundary, the concurrent case and the cancel path.

In scope to challenge: where the guard was placed, any behaviour change a user could notice, and whether this should have been a spec'd feature instead of a fix.

Engineering calls
  • Contract-free on purpose. There is no new field on records or entries, and no new RequestHost or WakeStore method. detectInterruptedRequests and createStaleRequestSweeper gain one optional option, wakeEntriesOf. The shipped sweeps fill it from the flow registry: the router's stale sweep, FlowState's startup sweep, and the route handlers' startup sweep.
  • Same threshold as the sweep, never shorter. This keeps a live request on a BullMQ peer reading as running, and leaves FIX-2135's busy refusal for a request that is parking untouched.
  • It composes with FIX-2170's requestStates. A crashed request whose heartbeat is stale reads parked (not running, not ended). So its held notice and its lapsed wait slots are released then. A notice it took still waits for the sweep's interrupted mark before another copy takes it over, as before.
  • No board or row I/O in the park-hook window. The change lives in engine reads and in the sweep. The sweep's extra I/O is wake reads and writes only.
  • The check-interrupted route does not bring wakes forward. It is a client-poked sweep. The periodic sweep that follows covers it.
  • The end-to-end test ends the task before the "crash", the way leg b's kill does: the ending is written, then the process dies before the notice is sent. The held notice is the same either way.
  • @flow-state-dev/core: JSDoc only, no changeset. @flow-state-dev/engine: patch changeset.
Everything else: evidence

Diagnosis. The closure's kept store (b-1-…sqlite) shows the following:

  • Writer's work request is in_progress, and its registry entry never beat (killed 2.3 s in; the heartbeat interval is 10 s).
  • After the restart, two onTaskSettled requests returned { owed: false }, and the wake in writer's session was still due.

A throwaway check on main showed the notice heard about 8 s after the request was marked interrupted, at the next resend: bounded, not lost.

Red then green. packages/orchestration/test/tasks/notice-after-crash.test.ts runs on in-memory stores and on SQLite:

  • Two engines share one store, with a 7 s threshold and a 500 ms sweep.
  • The first engine's filing turn is abandoned mid-run: its model call answers only the teardown's abort, and its heartbeat never beats again.

Case 1 (the sweep on):

  • Expects the notice heard at or after the threshold, within threshold + tick + 3 s, and heard once, and the request then marked interrupted.
  • Before the fix: expected 14059 to be less than or equal to 9500 (in-memory) and 14057 (SQLite). That is the backoff resend.
  • After the fix: heard about 7,050 ms after the last heartbeat (through the heartbeat read) or about 8,080 ms (through the sweep).

Case 2 (the sweep off):

  • Only the heartbeat read frees the notice. It is heard at the first resend past the threshold, and the record stays in_progress.
  • Before the fix it is never heard.

Both stores, 4 cases per run: 20 runs, 80/80 cases passed, both before and after merging main (FIX-2170).

Unit tests:

  • engine/test/wait-chain.test.ts: stale, fresh, at-threshold, queued, no-heartbeat and unregistered entries; heartbeats-off fallback.
  • engine/test/request-recovery.test.ts:
    • the session's wakes move forward, and another session's don't;
    • the version is bumped, so a pass in flight settles stale;
    • a request still beating keeps its wakes;
    • no entries named means no wake write;
    • a failing wake store still marks the request and logs.
  • engine/test/context/liveness-gate.test.ts: heartbeatStaleThreshold.

Checks run, after merging main:

  • pnpm typecheck, pnpm lint and pnpm fallow:ci.
  • node scripts/check-retired-terms.mjs.
  • node goals/vocabulary/one-word-per-thing/sweep.mjs.
  • Suites: engine, 4045 tests; orchestration, 2156 tests; workforce, 1602 tests.

Generated by Claude Code

…s the filing request

A notice held for its running filing request (FIX-2123) waited out its
resend backoff when that request's server died mid-run: the request still
read in_progress until the stale-request sweep marked it interrupted, and
nothing then sent the held notice before its next resend.

- waits.running answers false for an in_progress request whose registry
  heartbeat is silent past the host's stale-request threshold, where the
  flow's heartbeat is on and beats at least twice per threshold; otherwise
  the record alone answers, as before.
- The stale-request sweep and both startup sweeps bring the wakes kept for
  an interrupted request's session forward to now, so the filer's pass
  sends what it held at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
@linear-code

linear-code Bot commented Oct 24, 2026

Copy link
Copy Markdown

FIX-2166

@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 24, 2026 2:16pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 24, 2026 2:16pm UTC
flow-state-dev-ui Ignored Ignored Preview Oct 24, 2026 2:16pm UTC

Request Review

@jhoffner jhoffner left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stand-in review (Bugbot is out of usage) of head 57323b9.

Verdict. The src change is sound. The multi-process safety, wake versioning and per-entry early-fire checks all came out clean (details inline). There is one 🔴, and it is in the new e2e test: a status assertion that races the sweep and failed 4 of 26 cases locally.

What I ran (in a throwaway worktree):

  • notice-after-crash.test.ts 13× at head: 22/26 cases passed. The 4 failures were all expected 'in_progress' to be 'interrupted', and in every one of them arm 1 had delivered the notice at ~7,050 ms, before the sweep's next tick.
  • All engine/core src reverted to the merge base: red at the upper bound (14,061 / 14,073 ms against 10,500 ms). The test catches the bug.
  • Only wait-chain.ts reverted: green, ~8,080 ms in every run. The e2e pins arm 2 (bring the wakes forward). Arm 1 (the heartbeat read) is proven only by the unit test.
  • The engine unit tests (wait-chain, request-recovery, liveness-gate): 43/43 pass.

Findings

  • 🔴 notice-after-crash.test.ts:145: the interrupted status assertion races the sweep when arm 1 wins. Wait for the status, or drop the line.
  • 🟡 wait-chain.ts: the threshold is per process, not per deployment, and the stopped read stays on when sweeping is disabled. Make both deliberate in the JSDoc. Double delivery is ruled out by takeNotice and ended, which still reads the record only. The remaining cost is the notice running beside a live filer.
  • 🟡 request-wakes.ts: a pass that was already in flight can settle over the brought-forward wake, because move doesn't bump the version. That falls back to the backoff, so it isn't a regression. FIX-2150's re-owe is safe. Every entry pays only what is due, and waitDeadline is never moved.
  • 🟡 request-recovery.ts: optional lastHeartbeatAt == null guard. Also a minor divergence from the sweep on records that aren't org-attributed.
  • 🟡 test teardown: second.dispose() spends ~10 s draining in every case. There is also a glued comment on line 140.

Restraint and changeset. The engine patch is right: it adds an optional wakeEntriesOf on exported functions, pre-1.0, and the core change is JSDoc only. The README paragraph in orchestration was spliced into the middle of an existing line, which leaves one very long line; re-wrap it.


Generated by Claude Code

…rward, fail-safe heartbeat read

- notice-after-crash: wait for the sweep's interrupted mark instead of
  racing it; add a sweep-off case that pins the heartbeat read on its own;
  short router drain at teardown.
- bringSessionWakesForward owes from the wake it read (min due, version
  bump), so a pass in flight settles stale instead of undoing it.
- waits.running treats an entry with no heartbeat as alive; JSDoc says the
  threshold is per process and the read stays on with sweeping off.
- orchestration README paragraph re-wrapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
A run left hanging holds the process's request-end drain, which every
later host's shutdown in the same test worker waited on, timing out
afterEach hooks in other files. The never-answering model call now answers
the teardown's abort, and the teardown checks the run ended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
@jhoffner
jhoffner marked this pull request as ready for review October 24, 2026 14:07
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T14:18:16.443761Z c721154 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 87ce26b5bf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

waits.running read a live in_progress request as stopped when the
active-request registry lookup threw: the rejection reached the outer
catch, which answers false. Its notice was then sent, and its lapsed
slots released, while it still ran. The heartbeat lookup now fails safe
to the record's answer, as a missing entry or heartbeat does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VaN4uTiYqiabCbsFLWM52A
@jhoffner
jhoffner merged commit a46caac into main Oct 24, 2026
18 checks passed
@jhoffner
jhoffner deleted the fix/FIX-2166 branch October 24, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants